[Security Solution][Sourcerer] Replace Sourcerer with Discover Data View Picker #210585

lgestc · 2025-02-11T14:21:38Z

Unified Data View Picker: Phase 1 Implementation

Part of https://github.com/elastic/security-team/issues/11959

What This PR Does

This PR represents the first step in our transition from the current Sourcerer component to the new unified Data View Picker. Specifically, this implementation:

Creates a new Data View Picker component
Implements feature flag protection for all changes
Handles asynchronous effects through Redux listener middleware
Establishes a new Redux store architecture to support ad hoc data views infrastructure
Utilizes ad hoc data views to handle legacy patterns from series 7 (replacing the previous upgrade data view flow)

See the readme for more info:
x-pack/solutions/security/plugins/security_solution/public/data_view_manager/readme.md

What This PR Does NOT Cover

Does not affect screens other than Timelines
Does not modify the existing Sourcerer component in any way
Does not fully support all URL/local storage patterns

Implementation Notes

We've made several accommodations to support both Sourcerer and the new Data View Picker simultaneously during this transition period, including:

Some interfaces might look odd, especially the hooks that return the data view or patterns - this is intentional to support existing use cases
There are feature flag-based conditional statements throughout the code that will be removed once the transition is complete

Testing Instructions

Add the following feature flag to your configuration:

xpack.securitySolution.enableExperimental: ['newDataViewPickerEnabled']

Navigate to the Timelines interface
Test interactions with the new Data View Picker

Todo:

mention adhoc dataviews / their role in handling legacy timelines.
rename dataViewPicker to dataViewManager

WORK IN PROGRESS But reviews are welcome

x-pack/solutions/security/plugins/security_solution/public/data_view_picker/redux/types.ts

michaelolo24 · 2025-03-03T01:59:59Z

...tion/public/detections/components/alerts_table/timeline_actions/use_add_bulk_to_timeline.tsx

    browserFields,
    dataViewId,
    sourcererDataView,
    // important to get selectedPatterns from useSourcererDataView
    // in order to include the exclude filters in the search that are not stored in the timeline
    selectedPatterns,
  } = useSourcererDataView(scopeId);
+
+  if (newDataViewPickerEnabled) {
+    dataViewId = experimentalDataView.id ?? '';


can we default to an empty string if all the other parameters are provided in the reducer and not do these nullish checks everywhere the id is called?

eventually we will only pass the spec reference around, I imagine. Unfortunately this is how the DataViewSpec looks like 😭

michaelolo24 · 2025-03-03T02:01:55Z

...utions/security/plugins/security_solution/public/timelines/components/modal/header/index.tsx

+    const { dataView: experimentalDataView } = useDataView(DataViewPickerScopeName.timeline);
+    const experimentalBrowserFields = useBrowserFields(DataViewPickerScopeName.timeline);
+
+    if (newDataViewPickerEnabled) {


Thanks for doing it this way. I understand it's not ideal, but it keeps us safe-guarded for the time being 👍🏾

safer to do something like

const { browserFields: sourcererBrowserFields, sourcererDataView } = useSourcererDataView(SourcererScopeName.timeline); const { dataView: experimentalDataView } = useDataView(DataViewManagerScopeName.timeline); const experimentalBrowserFields = useBrowserFields(DataViewManagerScopeName.timeline); const finalBrowserFields = useMemo(() => newDataViewPickerEnabled ? experimentalBrowserFields : sourcererBrowserFields, [newDataViewPickerEnabled, experimentalBrowserFields, sourcererBrowserFields] ); const finalDataView = useMemo(() => newDataViewPickerEnabled ? experimentalDataView : sourcererDataView, [newDataViewPickerEnabled, experimentalDataView, sourcererDataView] );

let sometimes leads to non-obvious issues, and of all the props in kibana, browserFields is the one to be most defensive about

michaelolo24 · 2025-03-03T02:07:23Z

.../security/plugins/security_solution/public/timelines/components/timeline/tabs/esql/index.tsx

+  let [dataView, setDataView] = useState<DataViewSpec | undefined>();
+
+  if (newDataViewPickerEnabled) {
+    dataView = experimentalDataView;


I would just call setDataview(experimentalDataView) - if you're trying to avoid rerenders, then I would just use useRef instead.

.../security/plugins/security_solution/public/timelines/components/timeline/tabs/esql/index.tsx

michaelolo24 · 2025-03-03T02:08:41Z

...ecurity/plugins/security_solution/public/timelines/components/timeline/tabs/pinned/index.tsx

+  if (newDataViewPickerEnabled) {
+    selectedPatterns = experimentalSelectedPatterns;
+    sourcererDataView = experimentalDataView;
+    dataViewId = experimentalDataView.id ?? '';


same thing here re defaulting id to '' in the experimentalDataView initialization

vitaliidm · 2025-03-03T13:01:07Z

...lutions/security/plugins/security_solution/public/common/components/with_data_view/index.tsx

-    const dataView = useGetScopedSourcererDataView({
+    const experimentalDataView = useFullDataView({
+      dataViewPickerScope: DataViewPickerScopeName.timeline,
+    });


useFullDataView view called here and then dataView is getting replaced by experimentalDataView later only when newDataViewPickerEnabled is enabled.

It does look, that experimentalDataView is getting initiated every time, in disregard of feature flag.
Should it be avoided?

agree, avoiding use of let would achieve the same thing, and is imo a good rule of thumb to follow

…-fix'

elasticmachine · 2025-03-06T17:01:26Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 1cf669e

Failed CI Steps

FTR Configs #29

Test Failures

[job] [logs] FTR Configs #29 / lens app - group 5 lens drag and drop tests keyboard drag and drop should duplicate an element in a group

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	7056	7070	+14

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	8.9MB	8.9MB	+11.8KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`securitySolution`	85.5KB	85.5KB	+28.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`securitySolution`	577	579	+2

References to deprecated APIs

id	before	after	diff
`securitySolution`	351	355	+4

Total ESLint disabled count

id	before	after	diff
`securitySolution`	660	662	+2

History

💔 Build #281405 failed 069393d
💔 Build #281353 failed 33406cf

lgestc added 20 commits February 11, 2025 12:29

--wip-- [skip ci]

6a1d3e3

--wip-- [skip ci]

f26acdf

--wip-- [skip ci]

cc44f6c

--wip-- [skip ci]

d0c1970

--wip-- [skip ci]

99f434a

--wip-- [skip ci]

85d7233

--wip-- [skip ci]

73db16d

--wip-- [skip ci]

eadb7e6

--wip-- [skip ci]

a0b30c7

--wip-- [skip ci]

73ee667

--wip-- [skip ci]

df8d885

fix stephbeat

6cf58d6

--wip-- [skip ci]

c39eb3b

--wip-- [skip ci]

7020ef4

--wip-- [skip ci]

3d47008

--wip-- [skip ci]

83667ba

--wip-- [skip ci]

28aa4f0

--wip-- [skip ci]

f92eab5

--wip-- [skip ci]

0805163

--wip-- [skip ci]

aeeaf71

lgestc changed the title ~~Discover data view picker~~ WIP: Replace Sourcerer with Discover data view picker in the Timeline Feb 12, 2025

fix types

f49c643

elastic deleted a comment from elasticmachine Feb 14, 2025

lgestc and others added 7 commits February 14, 2025 11:54

Merge branch 'main' into discover_dp

1f4aebf

remove todo

63054f5

fixing tests

c7e0665

remove unused import

f261ab8

--wip-- [skip ci]

8c3e9a0

--wip-- [skip ci]

23439aa

--wip-- [skip ci]

770993c

michaelolo24 reviewed Mar 3, 2025

View reviewed changes

x-pack/solutions/security/plugins/security_solution/public/data_view_picker/redux/types.ts Outdated Show resolved Hide resolved

michaelolo24 reviewed Mar 3, 2025

View reviewed changes

.../security/plugins/security_solution/public/timelines/components/timeline/tabs/esql/index.tsx Show resolved Hide resolved

michaelolo24 reviewed Mar 3, 2025

View reviewed changes

lgestc and others added 4 commits March 3, 2025 11:56

Merge branch 'main' into discover_dp

38df1eb

remove middleware from data view picker

119d6a2

usecallback instead of usememo

6ff0437

fix shared loading state

d31acdc

vitaliidm reviewed Mar 3, 2025

View reviewed changes

lgestc added 5 commits March 4, 2025 09:11

naming

5ac74b7

naming

bac953a

naming

7b9b2df

naming

c96766b

update readme

f5beeca

lgestc force-pushed the discover_dp branch from 1981f86 to f5beeca Compare March 4, 2025 08:39

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

a6381fd

…-fix'

lgestc mentioned this pull request Mar 4, 2025

[Security Solution][Sourcerer] - Prevent setting update from solution component #201847

Closed

lgestc and others added 5 commits March 5, 2025 11:27

naming

b804725

remove eslint overrides

23f23d6

use isEmpty from lodash

b904a03

docs

8eb497e

Merge branch 'main' into discover_dp

33406cf

elastic deleted a comment from elasticmachine Mar 6, 2025

lgestc added 4 commits March 6, 2025 13:47

remove indicesExist from the data view manager api

082fc42

use isEmpty

069393d

cache full data view

ded2d6e

simplify useFullDataView api

1cf669e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Sourcerer] Replace Sourcerer with Discover Data View Picker #210585

[Security Solution][Sourcerer] Replace Sourcerer with Discover Data View Picker #210585

lgestc commented Feb 11, 2025 •

edited

Loading

michaelolo24 Mar 3, 2025

lgestc Mar 5, 2025

michaelolo24 Mar 3, 2025

kqualters-elastic Mar 5, 2025

michaelolo24 Mar 3, 2025

michaelolo24 Mar 3, 2025

vitaliidm Mar 3, 2025

kqualters-elastic Mar 5, 2025

elasticmachine commented Mar 6, 2025

ESLint disabled line counts

References to deprecated APIs

Total ESLint disabled count

[Security Solution][Sourcerer] Replace Sourcerer with Discover Data View Picker #210585

Are you sure you want to change the base?

[Security Solution][Sourcerer] Replace Sourcerer with Discover Data View Picker #210585

Conversation

lgestc commented Feb 11, 2025 • edited Loading

Unified Data View Picker: Phase 1 Implementation

What This PR Does

What This PR Does NOT Cover

Implementation Notes

Testing Instructions

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Mar 6, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Async chunks

Page load bundle

ESLint disabled line counts

References to deprecated APIs

Total ESLint disabled count

History

lgestc commented Feb 11, 2025 •

edited

Loading